Skip to content

perf(qwen3): skip redundant decode d2h/host copies in greedy path - #1

Open
lyfne123 wants to merge 5 commits into
chore/adapt-qwen3-decode-fwd-restorefrom
perf/qwen3-decode-skip-redundant-d2h
Open

perf(qwen3): skip redundant decode d2h/host copies in greedy path#1
lyfne123 wants to merge 5 commits into
chore/adapt-qwen3-decode-fwd-restorefrom
perf/qwen3-decode-skip-redundant-d2h

Conversation

@lyfne123

Copy link
Copy Markdown
Owner

Summary

Stacked on hw-native-sys#68 (needs its decode_fwd restore + pypto-lib bump). The fused Qwen3-14B decode kernel does device-side embedding and greedy sampling, so several per-step host copies are pure waste. This removes three, all host-only — no kernel change, no recompile. All Qwen entry points default to temperature=0 (greedy device sampling), so this is the hot path in normal runs.

# Copy eliminated Per decode step When
greedy logits d2h 16 × 152064 × 4B ≈ 9.7 MB greedy
next_hidden d2h 16 × 5120 × 2B ≈ 160 KB always
dead decode hidden copy [16, 5120] alloc + copy + float always

How

The runtime only device→host copies host shared-memory args; worker-resident DeviceTensor args stay on device. So:

  • ① logits — in the device-greedy path the token id comes from decode_sampled_ids_buffer (device), and host code never reads the full [BATCH, VOCAB] logits. Point the kernel's logits out slot at a reusable worker-resident DeviceTensor (_decode_device_scratch) instead of the shared host buffer. The kernel's internal argmax stays device-local; the d2h disappears. Non-greedy still uses the host buffer and returns real logits.
  • ② next_hidden — decode never returns it (run_decode always passes None to _integrated_sample_result), so its Out buffer is scratch. Route it to a device-resident buffer unconditionally.
  • ③ dead hidden_prepare_decode_inputs built a [B, hidden] bf16 buffer from batch.hidden_states (a zeros placeholder for device-embedding executors) that the kernel does not consume and no caller reads. Stop building/returning it.

DecodeResult.hidden_states / .logits become Optional; serving_worker reads host logits lazily (only on the non-greedy fallback), so None is safe. engine._sample_batch_rows already short-circuits on device-sampled ids before touching logits.

Verification (NPU, a2a3, greedy temperature=0)

  • Correct output: "The capital of France is""Paris … Washington", 24 tokens over 23 decode steps through the new path, no errors.
  • Run-to-run token divergence was confirmed to be inherent NPU non-determinism: the same unmodified binary run on two devices diverges identically at a near-tie argmax (step ~10). Not caused by this change — logits argmax reads the same values whether the buffer is host or device, and next_hidden is write-only.

Scope

  • Non-greedy / streaming paths unchanged (still return host logits).
  • Prefill's full-embedding h2d is not touched here — it needs kernel-side device gather.

ndleslx and others added 4 commits July 12, 2026 19:00
Add an end-to-end DeepSeek V4 HTTP completion test that starts the serving command with the documented TP=8 configuration, sends the prompt 'Huawei is' with greedy decoding, and requires the exact six-token completion ' a leading global provider of ICT'. Bound server startup and generation, preserve HTTP error response bodies, and print only a bounded server-log tail on failure.

Make process-group shutdown best-effort so stuck or failed waits cannot mask the original accuracy failure. Add focused CPU coverage for HTTP error diagnostics, bounded log reads, and the forced-kill timeout path.

Run the DeepSeek guard through task-submit with automatic eight-device allocation outside the configured whitelist, a 20-minute allocation wait, a 30-minute runtime limit, the CI model at /data/l00955553/model/dsv4-flash-w8a8, and the runtime settings required by the documented server configuration. Move both Qwen guards into their own automatic task-submit reservations and add a run-scoped marker so cancelled CI jobs can reap orphaned submissions.

Make serving workers close executor-owned runtime resources on every exit path, including initialization and inference failures, so distributed workers do not retain NPU memory after tests finish. Cover normal, exceptional, and repeated cleanup behavior with focused unit tests, and extend the NPU job timeout and dependencies for the HTTP guard.
ci: add DeepSeek V4 generation accuracy guard
…ecode-fwd-restore

Adapt Qwen3-14B serving to pypto-lib#747 (decode_layer -> decode_fwd rename)
The fused Qwen3-14B decode kernel does device-side embedding and greedy
sampling, so several per-step host copies are pure waste. Eliminate three,
all host-only (no kernel change, no recompile):

1. Greedy logits d2h. In the device-greedy path the token id is produced
   inside the kernel (decode_sampled_ids_buffer), so host code never reads
   the [BATCH, VOCAB] f32 logits. Point the kernel's logits `out` slot at a
   reusable worker-resident DeviceTensor instead of the shared host buffer;
   the runtime only d2h-copies host shared-memory args, so the ~9.7 MB/step
   copy (16 x 152064 x 4B) disappears. Non-greedy still uses the host buffer.

2. next_hidden d2h. Decode never returns next_hidden (run_decode always
   passes None to _integrated_sample_result), so its Out buffer is scratch.
   Route it to a device-resident buffer unconditionally, dropping a
   ~160 KB/step copy.

3. Dead decode hidden copy. _prepare_decode_inputs built a [B, hidden]
   bf16 buffer from batch.hidden_states (a zeros placeholder for
   device-embedding executors) that no caller reads and the kernel does not
   consume. Stop building/returning it.

DecodeResult.hidden_states / .logits become Optional; serving_worker reads
host logits lazily (only on the non-greedy fallback), so None is safe.

Verified on NPU (a2a3, greedy temperature=0): correct output
("The capital of France is" -> "Paris ... Washington"), 23 decode steps
through the new path, no errors. Run-to-run token divergence was confirmed
to be inherent NPU non-determinism (the same unmodified binary diverges
identically across devices at a near-tie argmax), not caused by this change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lyfne123
lyfne123 force-pushed the perf/qwen3-decode-skip-redundant-d2h branch from 9b66241 to 66976a7 Compare July 13, 2026 03:05
…gression)

The decode d2h/host-copy removal is numerically neutral: the kernel's
greedy argmax reads identical logit values whether the logits buffer is
host- or device-resident, next_hidden is write-only, and the removed
decode hidden copy was never consumed. Local reruns pass chunked-prefill
3/3 on this branch and 2/2 on main; the prior CI failure was an inherent
chunked-vs-full-prefill near-tie flip on the first token.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lyfne123
lyfne123 force-pushed the perf/qwen3-decode-skip-redundant-d2h branch from c85cabe to 94e8975 Compare July 13, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants